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DETAILED ACTION 

[1] Applicant's arguments received on July 18, 2008 pointed to certain deficiencies of the 
prior art cited in the Final Office Action mailed on April 18, 2008. In response to applicant's 
arguments, the following Office Action serves to replace the previous action and serves to clarify 
the examiner's position. A new shortened statutory time period of three (3) MONTHS and a 
new statutory period for reply is restarted to begin with the mailing date of this letter. The claim 
amendments after final was not be entered, and the following Office Action will address the 
claims as presented on January 22, 2008. 

Where for any reason it becomes necessaiy to remail any action (MPEP § 707.13), the action should be 
correspondingly redated, as it is the remailing date that establishes the beginning of the period for reply. Ex parte 
Gourtoff, 1924 CD. 153, 329 O.G. 536(Comm'r Pat. 1924). For Image File Wrapper (IFW) processing, see IFW 
Manual. 

MPEP 710.06 

A supplementary action after a rejection explaining the references more explicitly or giving the reasons more fully, 
even though no further references are cited, establishes a new date from which the statutory period runs. 

MPEP 710.06 

Amendments 

[2] This office action is responsive to the Claim Amendment and Applicant Arguments Made 
in an Amendment received on July 18, 2008 and Jan. 22, 2008. Claims 1-8 and 10 remain 
pending. 

Claim Objections 

[3] In response to applicant's claim objections amendments and remarks received on January 
22, 2008, the previous claim objections are withdrawn. 

Claim Rejections - 35 USC §101 
[4] In response to applicant's claim 35 USC § 101 rejection amendments and remarks 
received on January 22, 2008, the previous claim 35 USC § 101 rejections are withdrawn. 
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Claim Rejections - 35 USC § 103 

[5] The following is a quotation of 35 U.S.C. 103(a) which forms the basis for all 
obviousness rejections set forth in this Office action: 

(a) A patent may not be obtained though the invention is not identically disclosed or described as set forth in 
section 102 of this title, if the differences between the subject matter sought to be patented and the prior art are 
such that the subject matter as a whole would have been obvious at the time the invention was made to a person 
having ordinary skill in the art to which said subject matter pertains. Patentability shall not be negatived by the 
manner in which the invention was made. 

[6] Claims 1-2 and 10 are rejected under 35 U.S.C. 103(a) as being unpatentable over Ball 
Tracking and Virtual Replays for Innovative Tennis Broadcasts , 1 5 th International Conference on 
Pattern Recognition, 2000, Proceedings, Vol. 4, pg 152 - 156 {hereinafter "Pingali") in view of 
Pizano. 

Regarding Claim 1, while Pingali discloses an image recognition apparatus ("[a] general 
purpose computer" at p. 152, left column) for recognizing movements of players matched against 
each other (the tennis match involves at least two players) between domains partitioned with 
such an obstacle as net in a sport match or game (the tennis match must have a net and 
partitioned domains) from contents including a television program being telecasted to show the 
sport match or game ("live television broadcasts" at p. 152, left column), an image material in an 
uncompleted state for broadcasting and contents recorded in such a recording medium as a VTR 
(Introduction, p. 152; fig. 2), the image recognition apparatus comprising: 

a score information obtaining section ("[v]irtual [rjeplays and [v]isualtization" section at 
p. 154) configured to obtain score information ("includes score-based queries" at p. 154, right 
column; ALSO score information includes recorded paths of the tennis ball (e.g., fig. 3) ) 
indicative of scores ("includes score -based queries" at p. 154, right column; ALSO the path of 
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the ball is indicative of scores of the players) of the respective players which vary as the sport 
match or game proceeds; 

a play event information obtaining section ("[vjirtual [rjeplays and [v]isualtization" 
section at p. 154) configured to obtain play event information (play event information would 
comprising the path of the tennis ball, including specifically the exact location where the tennis 
ball hits the court) indicative of a characteristic movement (the path of the tennis ball is 
"indicative" of a characteristic movement of each of the players; i.e., a tennis ball path 
approaching the opposite side of the court indicates that a specific player hit it, a characteristic 
movement) of each of the players from picture information (e.g., fig. 3) included in the contents, 
the picture information containing images of the obstacle and the players (e.g., fig. 6); and 

an image substance recognizing section ("[vjirtual [r]eplays and [v]isualtization" section 
at p. 154) configured to make a comparison ( the comparison of the tennis ball position before 
and after it hits the ground must be made to determine the position where tennis ball hits the 
ground ) between a score information item obtained immediately before a point in time (e.g., the 
path of the tennis ball before hitting the ground at fig. 3) of generation of the play event 
information (e.g., the exact location where the ball hit the ground at fig. 3) and a score 
information item obtained immediately after the point in time (e.g., the path of the tennis ball 
after hitting the ground at fig. 3) and make reference to a result brought by the play event 
information (play event information would comprising the path of the tennis ball, including 
specifically the exact location where the tennis ball hits the court), thereby recognizing a 
substance (the substance being the exact location on the tennis court) of an image provided by 
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the play event information, Pingali does not teach wherein the score information itself is 
displayed on a screen. 

Pizano discloses an apparatus for detecting and interpreting textual captions in digital 
video signals (fig. 6) that teaches wherein the score information itself is displayed on a screen 
("3) SPORTS/TRANSPARENT" in fig. 1; the score in fig. 5). 

It would have been obvious to one of ordinary skill in the art at the time the invention 
was made for the screen of Pingali to include score information as taught by Pizano "to identify 
the collection of video frames that contain text captions.", Pizano, 5:54-56 and "to create a score 
browser which would enable a person to move directly to specific portions of the video", Pizano, 
1:61-63. 

Regarding Claim 2, while Pingali in view of Pizano disclose the image recognition 
apparatus according to Claim 1, Pingali in view of Pizano do not teach wherein the score 
information obtaining section is configured to obtain the score information from at least one of 
the picture information included in the contents, sound information including commentary voice 
of a commentator, and data information transmitted as multiplexed on radio waves during 
broadcasting. 

Pizano discloses an apparatus for detecting and interpreting texual captions in digital 
video signals (fig. 6; fig. 7; fig. 10) that teaches wherein a score information obtaining section 
(section responsible for fig. 6, fig. 7, fig. 10) is configured to obtain the score information 
(bottom pictures of the score in fig. 10) from at least one of the picture information (top picture 
in fig. 10) included in the contents (the contents being everything displayed at the top picture in 
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fig. 10), sound information including commentary voice of a commentator, and data information 
transmitted as multiplexed on radio waves during broadcasting. 

It would have been obvious to one of ordinary skill in the art at the time the invention 
was made for the score information obtaining section of Pingali in view of Pizano to include 
configuring to obtain the score information from at least one of the picture information included 
in the contents, sound information including commentary voice of a commentator, and data 
information transmitted as multiplexed on radio waves during broadcasting as taught by Pizano 
"to identify the collection of video frames that contain text captions.", Pizano, Col. 5, lines 54 - 
56 and "to create a score browser which would enable a person to move directly to specific 
portions of the video", Pizano, 1 :61-63. 

Regarding Claim 10, Claim 1 recites identical features as in Claim 10. Thus, 
references/arguments equivalent to those presented above for Claim 1 are equally applicable to 
Claim 10. 

[7] Claims 3-8 are rejected under 35 U.S.C. 103(a) as being unpatentable over Pingali in 
view of Pizano and Automatic Classification of Tennis Video for High-level Content-based 
Retrieval , Proceedings of the 1998 International Workshop on Content-Based Access of Image 
and Video Databases (CAIVD '98), 1998, pp 81 - 90 {hereinafter "Sudhir"). 

Regarding Claim 3, while Pingali in view of Pizano disclose image recognition 
apparatus according to Claims 1 or 2, further comprising: 

a domain item extracting section ("[vjirtual [rjeplays and [v]isualtization" section at p. 
154) configured to extract instrument information on an instrument moving between the domains 
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to serve as an object of score count in the sport match or game (an ace by Sampras in fig. 3 was 
within the tennis boundary domain, thus a scoring count) 

Pingali in view of Pizano does not teach a domain item extracting section configured to 
extract from the picture information facility information including information on the obstacle, 
information on the domains and information on boundary lines between the domains and an area 
outside the domains, and player's position information indicative of a player's position; rule 
information storage section configured to store rule information on the sport match or game; and 
basic movement storage section configured to store basic movement information on players' 
characteristic movements generalized in a sport of concern, wherein the play event information 
obtaining section includes a play event information determining section configured to determine 
a play event information item on a play event characteristic of each of the players included in the 
picture information as the play event information based on domain items extracted from the 
picture information, the rule information, and the basic movement information stored in the basic 
movement information storage section. 

Sudhir discloses an automatic classification of tennis video for high-level content-based 
retrieval (Introduction, pg 81) that teaches 

a domain item extracting section (fig. 3; Table 2) configured to extract from the picture 
information facility information including information on the obstacle (dashed line in fig. 7), 
information on the domains and information on boundary lines between the domains and an area 
outside the domains (fig. 7), and player's position information indicative of a player's position 
("Player Tracking Module" in fig. 1); 
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rule information storage section ("court-line detection module" in fig. 1 for storing rules 
on court-line boundaries) configured to store rule information on the sport match or game; and 

basic movement storage section ("Player Tracking Module" in fig. 1) configured to store 
basic movement information on players' characteristic movements generalized in a sport of 
concern, wherein a play event information obtaining section ("High-level Reasoning module" in 
deciding plays like "high-level events like baseline-rallies, passing-shots, serve-and-volleying 
and net-games" in s. 9, p 89) includes a play event information determining section configured to 
determine a play event information item on a play event characteristic of each of the players 
included in the picture information as the play event information based on domain items (the 
boxes outlining the players on the right side of fig. 6 arc the domains) extracted from the picture 
information, the rule information (fig. 7), and the basic movement information stored in the basic 
movement information storage section (fig. 1). 

It would have been obvious to one of ordinary skill in the art at the time the invention 
was made for the apparatus of Pingali in view of Pizano to include a domain item extracting 
section configured to extract from the picture information facility information including 
information on the obstacle, information on the domains and information on boundary lines 
between the domains and an area outside the domains, and player's position information 
indicative of a player's position; rule information storage section configured to store rule 
information on the sport match or game; and basic movement storage section configured to store 
basic movement information on players' characteristic movements generalized in a sport of 
concern, wherein the play event information obtaining section includes a play event information 
determining section configured to determine a play event information item on a play event 
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characteristic of each of the players included in the picture information as the play event 
information based on domain items extracted from the picture information, the rule information, 
and the basic movement information stored in the basic movement information storage section as 
taught by Sudhir as "there is dire need for algorithms that are able to automatically infer high- 
level content from data.", Sudhir, Introduction, p 81. 

Regarding Claim 4, while Pingali in view of Pizano and Sudhir disclose the apparatus 
according to Claim 3, Pingali in view of Pizano and Sudhir do not teach wherein the player's 
position information is position information indicative of a domain containing each of the 
players and the instrument constantly held and used by the player. 

Sudhir discloses an automatic classification of tennis video for high-level content-based 
retrieval (Introduction, pg 81) that teaches wherein the player's position information is position 
information indicative of a domain containing each of the players and the instrument constantly 
held and used by the player (the player, the tennis racquet, and tennis ball at the point of impact 
are all in the boxes (domain) on the right side of fig. 6). 

It would have been obvious to one of ordinary skill in the art at the time the invention 
was made for the apparatus of Pingali in view of Pizano and Sudhir to include wherein the 
player's position information is position information indicative of a domain containing each of 
the players and the instrument constantly held and used by the player as taught by Sudhir as 
"there is dire need for algorithms that are able to automatically infer high-level content from 
data.", Sudhir, Introduction, p 81. 

Regarding Claim 5, Pingali discloses wherein the domain item extracting section 
("[v]irtual [r]eplays and [v]isualtization" section at p. 154) is configured to extract the player's 
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position information (the origination of the path of the tennis ball in fig. 2 and fig. 4 is the 
player's position information) from the picture information (fig. 4) based on the facility 
information (coordinates to track/detect/match instruments, obstacle, and boundaries in fig. 4 and 
fig. 2) extracted by the domain item extracting section. 

Regarding Claim 6, Pingali discloses wherein the domain item extracting section 
("[v]irtual [rjeplays and [visualization" section at p. 154) is configured to extract the instrument 
information from the picture information (fig. 2, fig. 4, path of the tennis ball) based on the 
facility information (coordinates to track/detect/match instruments, obstacle, and boundaries in 
fig. 4 and fig. 2) and the player's position information (the origination of the path of the tennis 
ball in fig. 2 and fig. 4 is the player's position information) extracted by the domain item 
extracting section. 

Regarding Claim 7, while Pingali in view of Pizano and Sudhir disclose the apparatus 
according to Claim 3, Pingali in view of Pizano and Sudhir do not teach wherein the play event 
information obtaining section includes a play event index information output section configured 
to output plural play event information items determined by the play event information 
determining section as arranged in a time sequence. 

Sudhir discloses an automatic classification of tennis video for high-level content-based 
retrieval (Introduction, p. 81) that teaches wherein the play event information obtaining section 
includes a play event index information output section (Table 3, p. 88) configured to output 
plural play event information items ("High-level Annotation" column in Table 3) determined by 
the play event information determining section as arranged in a time sequence (the table suggests 
the arrangement of time sequence). 
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It would have been obvious to one of ordinary skill in the art at the time the invention 
was made for the apparatus of Pingali in view of Pizano and Sudhir to include wherein the play 
event information obtaining section includes a play event index information output section 
configured to output plural play event information items determined by the play event 
information determining section as arranged in a time sequence as taught by Sudhir as "there is 
dire need for algorithms that are able to automatically infer high-level content from data.", 
Sudhir, Introduction, p 8 1 . 

Regarding Claim 8, while Pingali in view of Pizano and Sudhir disclose the apparatus 
according to Claim 3, Pingali in view of Pizano and Sudhir do not teach wherein the play event 
index information output section is configured to output the play event information items 
together with instrument information items in a time sequence. 

Sudhir discloses an automatic classification of tennis video for high-level content-based 
retrieval (Introduction, p. 81) that teaches wherein the play event index information output 
section (Table 3, p. 88) is configured to output the play event information items ("High-level 
Annotation" column in Table 3) together with instrument information items ("BL" is "Baseline", 
and thus the table indicates the location information from which the instrument was during the 
serve) in a time sequence (the table suggests the arrangement of time sequence). 

It would have been obvious to one of ordinary skill in the art at the time the invention 
was made for the apparatus of Pingali in view of Pizano and Sudhir to include wherein the play 
event index information output section is configured to output the play event information items 
together with instrument information items in a time sequence as taught by Sudhir as "there is 
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dire need for algorithms that are able to automatically infer high-level content from data.", 
Sudhir, Introduction, p. 81. 

Response to Arguments 

[9] Applicant's arguments filed on January 22, 2008 with respect to Claim 1 has been 
respectfully and fully considered, but unpersuasive in view of the new ground of rejection in the 
corrected Final Office Action. 
Summary of Remarks regarding Claim 1 

Applicant argues that from fig. 1 and the corresponding description, the second item of 
claim 1 has not been disclosed (i.e. the "play even information obtaining section"). Pingali 
discusses a trigger event, but does not mention the italicized part of item 2 (i.e. "from picture 
information included in the contents"). That is, nowhere does Pingali disclose or fairly suggest 
that the trigger event comes from picture information. To the contrary, it seems as though 
Pingali must wait for a trigger input based on something other than picture information; perhaps 
a human input. Once this happens then Pingali "executes a loop of capturing frames from the 
camera pair. ..." Thus, before the trigger event, Pingali does not teach capturing frames. Thus, 
no picture information can be obtained before the triggering event. As such, it appears that the 
triggering event cannot be based on picture information as required by Claim 1 . (Applicant 
Resp. at 12, Jan. 22, 2008.) 

Claim 1 specifically requires an " image substance recognizing section" (emphasis added). 
Keeping track of a changing game score is not recognizing the substance of an image. The 
Examiner must point to how an image is being recognized. Applicant respectfully submits that 
that this claimed feature is not disclosed or fairly suggested in Pingali. (Resp. at 13.) 
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Furthermore, the substance of the image must be "provided by the play event 
information," as required by Claim 1. The play event information, as indicated above contains 
picture information. As required in Claim 1, this picture information may be in the form of "score 
information displayed on a screen." Claim 1 requires the "image substance recognizing section" 
to recognize an image (i.e. a score), that is displayed on a screen. Nowhere is it apparent that 
Pingali discloses or fairly suggests this feature. As Pingali is not concerned with image 
recognition, but appears to be dealing with image tracking, there is no reason for Pingali to deal 
with an "image recognition section." (Resp. at 13.) 
Examiner's Response regarding Claim 1 

Applicant's arguments with respect to claim 1 has been considered but are unpersuasive 
in view of the new ground of rejection ("Virtual Replays and Visualization" section of Pingali). 
See section 10, Examiner's Response regarding Claim 1 for further detail in response to 
Applicant's arguments filed on July 18, 2008 below. 

[10] Applicant's arguments filed on July 18, 2008 with respect to Claim 1 has been 
respectfully and fully considered, and not found persuasive. 
Summary of Remarks regarding Claim 1 

Regarding the second element of claim 1 , as labeled above, Applicant notes that the play 
event information obtaining section of claim 1 obtains information "indicative of a characteristic 
movement of each of the players." In Pingali, the disclosure seems to only be concerned with 
"ball tracking." That is, Applicant respectfully submits that Pingali does not disclose obtaining 
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movement information from each of the players. As such, this feature of independent claim 1 is 
not disclosed or fairly suggested by the cited references. 

Regarding the third element of claim 1, the "image substance recognition section," the 
Examiner maintains his position. However, as indicated in the response dated January 22, 2008, 
it does not appear that the cited references disclose the claimed invention. Specifically, claim 
requires a "comparison" step between score information obtained immediately before and after 
play event information. 

While the Examiner believes the references disclose a play event information obtaining 
section, and an image substance recognizing section, there does not appear to be any support 
comparing two images to one another and determining the substance of an image provided by 
the play event information based on said comparison. (Applicant Resp. at 4-5, July 7, 2008.) 
Examiner's Response regarding Claim 1 

Regarding the first element of claim 1 , the claim language is broad enough to include the 
ball tracking feature of Pingali. The ball tracking feature of Pingali includes recorded scores and 
paths of balls throughout the match. The exact ball path and contact with the court is indicative 
of scores of the respective players (e.g., fig. 3). The ball hitting within or outside the marked 
bounds is indicative of a score. The Examiner suggests amending the first element such that 
there is a clear definitive description that the actual score is being read. 

Regarding the second element of claim 1, the claim language is broad enough to include 
the ball tracking feature of Pingali. Information including the path and location where the ball 
hits the court is play event information. This play event information is again indicative of a 
characteristic movement of the players. A ball being projected into the opposite side of the court 
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is indicative that a particular player hit it, which is a characteristic movement of the player. The 
Examiner suggests amending the first element such that there is a clear definitive description that 
information is being recognized from the player's actual movement (e.g., a volley). 

The Examiner notes that the argument of "comparing two images to one another" for the 
play event information obtaining section is not in the claim language. The claim is broad enough 
to not include such a limitation, which Pingali appears not the anticipate. 

The Examiner further notes that the score information obtaining section and play event 
information obtaining section is nothing more than a recording of a tennis match. A recorded 
tennis match would inherently include "score information indicative of scores", "play event 
information indicative of a characteristic movement of each of the players", and "score 
information" (emphasis added). Furthermore, a recording of the path of a tennis ball which 
correlate directly with the score of the match reads on an image substance recognizing section. 
The Examiner suggests limitations to acquiring the actual score information, and using that 
actual score information to compare definitive score information items. 

Conclusion 

[8] The prior art made of record and not relied upon is considered pertinent to applicant's 
disclosure. 

(i) Gong et al, Automatic Parsing of TV Soccer Programs, Proc. IntT Conf. On 
Multimedia Computing and Systems, pp. 167-168, 1997; 

(ii) Saur et al, Automated Analysis and Annotation of Basketball Video , Store and 
Retrieval for Image and Video Databases V, SPIE Vol. 3022, pp. 176-187, 1997; and 
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(iii) Sudhir et al, Automatic Classification of Tennis Video for High-Level Content- 
Based Retrieval , Proc. Of IEEE Workshop on Content-Based Access of Image and Video 
Databases, CAIVD'98, 1998. 
[11] THIS ACTION IS MADE FINAL. Applicant is reminded of the extension of time 
policy as set forth in 37 CFR 1.136(a). 

A shortened statutory period for reply to this final action is set to expire THREE 
MONTHS from the mailing date of this action. In the event a first reply is filed within TWO 
MONTHS of the mailing date of this final action and the advisory action is not mailed until after 
the end of the THREE-MONTH shortened statutory period, then the shortened statutory period 
will expire on the date the advisory action is mailed, and any extension fee pursuant to 37 
CFR 1 .136(a) will be calculated from the mailing date of the advisory action. In no event, 
however, will the statutory period for reply expire later than SIX MONTHS from the mailing 
date of this final action. 

[9] Any inquiry concerning this communication or earlier communications from the 
examiner should be directed to DAVID P. RASHID whose telephone number is (571)270-1578. 
The examiner can normally be reached Monday - Friday 7:30 - 17:00 ET. 

If attempts to reach the examiner by telephone are unsuccessful, the examiner's 
supervisor, Vikkram Bali can be reached on (571) 272-74155. The fax phone number for the 
organization where this application or proceeding is assigned is 571-273-8300. 

Information regarding the status of an application may be obtained from the Patent 
Application Information Retrieval (PAIR) system. Status information for published applications 
may be obtained from either Private PAIR or Public PAIR. Status information for unpublished 
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applications is available through Private PAIR only. For more information about the PAIR 

system, see http://pair-direct.uspto.gov. Should you have questions on access to the Private PAIR 

system, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would 

like assistance from a USPTO Customer Service Representative or access to the automated 

information system, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000. 

/ David P. Rashid/ 
Examiner, Art Unit 2624 

David P Rashid 
Examiner 
Art Unit 26244 



/Vikkram Bali/ 

Supervisory Patent Examiner, Art Unit 2624 



